Partially Observable Markov Decision Processes for Spoken Dialogue Management

Partially Observable Markov Decision Processes for Spoken Dialogue Management Jason D. Williams The design of robust spoken dialog systems is a significant research challenge. Speech recognition errors are common and hence the state of the conversation can never be known with certainty, and users can react in a variety of ways making deterministic forward planning impossible. This thesis argues that a partially observable Markov decision process (POMDP) provides a principled formalism for modelling human-machine conversation. Further, this thesis introduces the SDS-POMDP framework which enables statistical models of users’ behavior and the speech recognition process to be combined with handcrafted heuristics into a single framework that supports global optimization. A combination of theoretical and empirical studies confirm that the SDS-POMDP framework unifies and extends existing techniques, such as local use of confidence score, maintaining parallel dialog hypotheses, and automated planning. Despite its potential, the SDS-POMDP model faces important scalability challenges, and this thesis next presents two methods for scaling up the SDS-POMDP model to realistically sized spoken dialog systems. First, summary point-based value iteration (SPBVI) enables a single slot (a dialog variable such as a date, time, or location) to take on an arbitrary number of values by restricting the planner to consider only the likelihood of the best hypothesis. Second, composite SPBVI (CSPBVI) enables dialog managers consisting of many slots to be created by planning locally within each slot, and combining these local plans into a global plan using a simple heuristic. Results from dialog simulation show that these techniques enable the SDS-POMDP model to handle real-world dialog problems while continuing to out-perform established techniques and hand-crafted dialog managers. Finally, application to a real spoken dialog system is demonstrated.